Radar apparatus and method for classifying object

ABSTRACT

A radar apparatus installed in a vehicle includes a transceiver that transmits a radar signal to an outside of the vehicle and receives a radar signal reflected from an object; a signal processing unit that processes the reflected radar signal to detect the object; a fusion data generation unit that generates fusion data based on radar data and camera data; and a classification unit that classifies the detected object using an artificial intelligence module trained based on the generated fusion data.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims the benefit under 35 USC 119(a) of Korean PatentApplication No. 10-2020-0038964 filed on 31 Mar. 2020 and Korean PatentApplication No. 10-2021-0035112 filed on 18 Mar. 2021, in the KoreanIntellectual Property Office, the entire disclosures of which areincorporated herein by reference for all purposes.

TECHNICAL FIELD

The present disclosure relates to a radar apparatus and method forclassifying an object.

BACKGROUND

An automotive radar functions to provide a driver with locationinformation of nearby vehicles and obstacles present in front of, besideand behind the driver's vehicle.

Such an automotive radar has a high range accuracy in detecting anobject and is not affected by driving conditions of the vehicle (forexample, bad weather, night time, etc.) and thus provides stable objectdetection performance. Also, the automotive radar can measure the speedof an object and thus can distinguish whether the object is moving orstationary.

However, the automotive radar has a lower horizontal resolution than anautomotive camera or a Lidar. Thus, it is difficult to estimate andclassify the shape of an object using the automotive radar. Also, in anannotation process for obtaining ground truth used for artificialintelligence training of the automotive radar, it is difficult for thedriver to identify an object based only on the measurement result (forexample, location and speed) of the object detected by the automotiveradar. Therefore, without additional information from other sensors, itis difficult to classify the object detected by the automotive radar.

Meanwhile, the automotive camera has a higher horizontal resolution thanthe automotive radar and thus has been useful to recognize and classifyan object.

However, the automotive camera is capable of measuring the distance toan object by means of stereo vision and the like, but has a low rangeaccuracy and is not suitable for detecting an object at a remotedistance. Also, the automotive camera is affected by driving conditionsof the vehicle and thus provides degraded object detection performancein bad weather or at night time.

PRIOR ART DOCUMENT

-   Patent Document 1: Korean Patent Laid-open Publication No.    2020-0132137 (published on Nov. 25, 2020)

SUMMARY

In view of the foregoing, the present disclosure provides a techniquefor generating fusion data based on radar data and camera data andclassifying a detected object through artificial intelligence modulestrained based on the generated fusion data.

The problems to be solved by the present disclosure are not limited tothe above-described problems. There may be other problems to be solvedby the present disclosure

According to at least one example embodiment, a radar apparatusinstalled in a vehicle may include a transceiver that transmits a radarsignal to an outside of the vehicle and receives a radar signalreflected from an object; a signal processing unit that processes thereflected radar signal to detect the object; a fusion data generationunit that generates fusion data based on radar data and camera data; anda classification unit that classifies the detected object using anartificial intelligence module trained based on the generated fusiondata.

According to at least one other example embodiment, a method forclassifying an object by a radar apparatus installed in a vehicle, mayinclude transmitting a radar signal to an outside of the vehicle;receiving a radar signal reflected from an object; processing thereflected radar signal to detect the object; generating fusion databased on radar data and camera data; and classifying the detected objectusing an artificial intelligence module trained based on the generatedfusion data.

This summary is provided by way of illustration only and should not beconstrued as limiting in any manner. Besides the above-describedexemplary embodiments, there may be additional exemplary embodimentsthat become apparent by reference to the drawings and the detaileddescription that follows.

According to any one of the above-described embodiments of the presentdisclosure, it is possible to generate fusion data based on radar dataand camera data and classify a detected object through artificialintelligence modules trained based on the generated fusion data.

Also, according to the present disclosure, the artificial intelligencemodule can be trained in real time by using the fusion data, and, thus,the trained artificial intelligence module can improve object detectionand classification performance.

Further, according to the present disclosure, it is possible to generatefusion data based on data from a radar that provides stable objectdetection performance even in bad weather or at night time and improveobject detection performance of an artificial intelligence camera modulebased on the fusion data.

Furthermore, according to the present disclosure, a radar apparatus canstably detect and classify an object using the artificial intelligencemodules trained based on the fusion data even in a situation, such asbad weather or night time, where it is difficult for a camera toidentify an object.

Moreover, according to the present disclosure, the radar apparatus canuse the artificial intelligence modules trained based on the fusion datato later classify an object, which has been detected by the radarapparatus, without using a camera.

BRIEF DESCRIPTION OF THE DRAWINGS

In the detailed description that follows, embodiments are described asillustrations only since various changes and modifications will becomeapparent to those skilled in the art from the following detaileddescription. The use of the same reference numbers in different figuresindicates similar or identical items.

FIG. 1 is a block diagram of a radar apparatus according to anembodiment of the present disclosure.

FIG. 2A is diagram for explaining data used to generate fusion dataaccording to an embodiment of the present disclosure.

FIG. 2B is diagram for explaining data used to generate fusion dataaccording to an embodiment of the present disclosure.

FIG. 2C is diagram for explaining data used to generate fusion dataaccording to an embodiment of the present disclosure.

FIG. 3 is a diagram for explaining a process for generating fusion dataaccording to an embodiment of the present disclosure.

FIG. 4 is a diagram for explaining a process for training an artificialintelligence module according to an embodiment of the presentdisclosure.

FIG. 5 is a diagram for explaining a method for generating fusion datadepending on a driving condition of a vehicle according to an embodimentof the present disclosure.

FIG. 6 is a diagram for explaining a method for generating fusion datadepending on performance of an artificial intelligence radar moduleaccording to an embodiment of the present disclosure.

FIG. 7 is a flowchart showing a method for classifying an object by aradar apparatus installed in a vehicle according to an embodiment of thepresent disclosure.

DETAILED DESCRIPTION

Hereafter, example embodiments will be described in detail withreference to the accompanying drawings so that the present disclosuremay be readily implemented by those skilled in the art. However, it isto be noted that the present disclosure is not limited to the exampleembodiments but can be embodied in various other ways. In the drawings,parts irrelevant to the description are omitted for the simplicity ofexplanation, and like reference numerals denote like parts through thewhole document.

Throughout this document, the term “connected to” may be used todesignate a connection or coupling of one element to another element andincludes both an element being “directly connected” another element andan element being “electronically connected” to another element viaanother element. Further, it is to be understood that the term“comprises or includes” and/or “comprising or including” used in thedocument means that one or more other components, steps, operationand/or the existence or addition of elements are not excluded from thedescribed components, steps, operation and/or elements unless contextdictates otherwise; and is not intended to preclude the possibility thatone or more other features, numbers, steps, operations, components,parts, or combinations thereof may exist or may be added.

Throughout this document, the term “unit” includes a unit implemented byhardware and/or a unit implemented by software. As examples only, oneunit may be implemented by two or more pieces of hardware or two or moreunits may be implemented by one piece of hardware. However, the “unit”is not limited to the software or the hardware and may be stored in anaddressable storage medium or may be configured to implement one or moreprocessors. Accordingly, the “unit” may include, for example, software,object-oriented software, classes, tasks, processes, functions,attributes, procedures, sub-routines, segments of program codes,drivers, firmware, micro codes, circuits, data, database, datastructures, tables, arrays, variables and the like. The components andfunctions provided by the “units” can be combined with each other or canbe divided up into additional components. Further, the components andthe “units” may be configured to implement one or more CPUs in a deviceor a secure multimedia card.

In the present specification, some of operations or functions describedas being performed by a device may be performed by a server connected tothe device. Likewise, some of operations or functions described as beingperformed by a server may be performed by a device connected to theserver.

Hereinafter, embodiments of the present disclosure will be explained indetail with reference to the accompanying drawings.

FIG. 1 is a block diagram of a radar apparatus 10 according to anembodiment of the present disclosure.

Referring to FIG. 1 , the radar apparatus 10 may include a transceiver100, a signal processing unit 110, a fusion data generation unit 120, aclassification unit 130, a training unit 140 and an autonomous drivingunit 150. However, the radar apparatus 10 illustrated in FIG. 1 is justone of embodiments of the present disclosure and can be modified invarious ways based on the components illustrated in FIG. 1 .

The transceiver 100 may transmit a radar signal to the outside of avehicle and receive a radar signal reflected from an object. Forexample, the object may include an obstacle, a moving body, apedestrian, etc. located in front of, beside and behind the vehicle.

The signal processing unit 110 may process the reflected radar signal todetect the object. Here, the reflected radar signal is received whilethe vehicle equipped with the radar apparatus 10 is driving.

Specifically, the signal processing unit 110 may perform a signalprocessing to the reflected radar signal to detect the object. Forexample, referring to FIG. 2A, the signal processing unit 110 mayextract a plurality of point cloud data 201 (small boxes) constitutingat least one object based on the reflected radar signal, and derive adetection result value of the object by using the extracted plurality ofpoint cloud data 201. Here, the detection result value of the object mayinclude location information and speed information of the object andangle information between the object and the vehicle (vehicle equippedwith the radar apparatus 10) derived based on the plurality of pointcloud data 201.

The signal processing unit 110 may recognize the object based on thedetection result value of the object. The signal processing unit 110 maydetermine whether or not the detected object is a real object based onthe detection result value of the object, and derive a recognitionresult value of the object based on the determination. For example,referring to FIG. 2A, the signal processing unit 110 may cluster theplurality of point cloud data 201, estimate a clustered point cloud set203 (large square box) as a real object and regard the clustered pointcloud set 203 as a recognition result value of the object. Further, thesignal processing unit 110 may track the clustered point cloud set andderive the tracking result as a recognition result value of the object.Furthermore, the signal processing unit 110 may correct a moving speedof the real object corresponding to the clustered point cloud setdepending on whether the vehicle is moving and derive the correctedresult as a recognition result value of the object.

The signal processing unit 110 may generate radar data based on a fastFourier transform value of the reflected radar signal, the detectionresult value of the object and the recognition result value of theobject.

An image processing unit (not shown) may detect and classify the objectbased on image data received from a camera installed in the vehicle. Forexample, referring to FIG. 2B, the image processing unit (not shown) mayinput the image data into an artificial intelligence camera module 20and then derive camera data including a detection result value (forexample, location information of the object, etc.) and a classificationresult value (for example, vehicle type information of the object)derived by the artificial intelligence camera module 20. Here, the imageprocessing unit (not shown) may recognize the object from the image datathrough the artificial intelligence camera module 20 and set a boundingbox for the recognized object to search for the location of therecognized object. Further, the image processing unit (not shown) maydetermine which of a plurality of categories (for example, truck, bus,sedan, etc.) the object (for example, vehicle) recognized through theartificial intelligence camera module 20 belongs to and then classifythe recognized object into the corresponding category.

The fusion data generation unit 120 may generate fusion data based onthe radar data and the camera data.

Herein, the radar data may include data derived from a radar signal,such as a fast Fourier transform value of the radar signal reflectedfrom the object, a detection result value of the object, a recognitionresult value of the object and the like. Herein, the camera data may bederived through the artificial intelligence camera module and mayinclude a detection result value of an object included in image datagenerated by the camera installed in the vehicle and a classificationresult value of the object.

The fusion data generation unit 120 may project the camera data to aradar coordinate system of the radar apparatus 10 and match the cameradata and the radar data projected to the radar coordinate system foreach target to generate fusion data. For example, referring to FIG. 2C,the fusion data generation unit 120 may transform the locationinformation of the object included in the camera data into a radarcoordinate system of the radar apparatus 10 installed in a vehicle 205(for example, an XYZ coordinate system around the vehicle 205) and checkwhether the coordinates of the object included in the camera datatransformed into the radar coordinate system are similar to thecoordinates of the object included in the radar data. In this case, thetransformation into the radar coordinate system may be performed tocompensate for a location error if there is a location error between thecamera data and the radar data. Then, if the coordinates of the objectincluded in the camera data transformed into the radar coordinate systemare similar to the coordinates of the object included in the radar data,the fusion data generation unit 120 may recognize the object included inthe camera data and the object included in the radar data as the sameone and match each other.

Meanwhile, if the location error between the camera data and the radardata is less than a threshold value, the fusion data generation unit 120does not project the camera data to the radar coordinate system of theradar apparatus 10, but may match the camera data and the radar data foreach target to generate fusion data.

The fusion data may be used to obtain a classification result with highreliability through mutual matching between the camera data and theradar data and cumulative statistics over time.

The advantage of the radar apparatus 10 is that it has high range andvelocity accuracy and can quickly derive a detection result. Theadvantage of the camera is that it has high horizontal resolution andcan distinguish the types of vehicles (for example, truck, bus, sedan,motorcycle, etc.). According to the present disclosure, it is possibleto generate fusion data with high accuracy for an object detected by thecamera and the radar apparatus 10 by using the advantages of the cameraand the radar apparatus 10 and complementing the disadvantages thereof.

FIG. 3 is a diagram for explaining a process for generating fusion data.

Referring to FIG. 3 , the signal processing unit 110 may generate radardata based on a radar signal reflected from an object. The imageprocessing unit (not shown) may generate camera data based on image datathrough the artificial intelligence camera module 20. Here, the cameradata may be coordinate-transformed into a radar coordinate system so asto be matched with the radar data for each target. The fusion datageneration unit 120 may generate fusion data using the radar data andthe camera data transformed into the radar coordinate system.

Meanwhile, the fusion data generation unit 120 may generate fusion databy analyzing driving conditions of a vehicle while the vehicle isdriving and giving weightings to the camera data and the radar databased on the analyzed driving conditions. Here, the driving conditionsrefers to external environmental factors that affect the vehicle whilethe vehicle is driving and may include various types of drivingcondition information (for example, rainy weather, foggy weather, nightdriving, etc.). For example, referring to FIG. 5 , if it is rainy (orfoggy) when the vehicle is driving, the accuracy in detecting an object(for example, vehicle) using camera data 50 decreases. Therefore, inthis case, the fusion data generation unit 120 may generate fusion databy giving a higher weighting to radar data 52 than to the camera data50. Here, the fusion data generated by giving a higher weighting to theradar data 52 may be used for additional training by the training unit140, and location information of the object may be estimated from thecamera data 50 based on location information of the object included inthe radar data 52.

Meanwhile, the training unit 140 may input radar data into an artificialintelligence radar module of the radar apparatus 10 to train theartificial intelligence radar module, and input image data into anartificial intelligence camera module of the camera to train theartificial intelligence camera module. Herein, the radar data may begenerated while the vehicle is driving based on a reflected radar signalreceived while the vehicle is driving, and the image data may begenerated while the vehicle equipped with the camera is driving.

The training unit 140 may input the radar data into the artificialintelligence radar module and train the artificial intelligence radarmodule so that the artificial intelligence radar module can perform adetection process and a classification process with respect to anobject.

The training unit 140 may input the image data into the artificialintelligence camera module and train the artificial intelligence cameramodule so that the artificial intelligence camera module can derivecamera data including a detection result value of the object and aclassification result value of the object.

The radar apparatus and the camera installed in the vehicle may receivea radar signal while the vehicle is driving and generate image data inreal time. Therefore, the camera data and the radar data used in thetraining unit 140 may be generated in real time based on the radarsignal and the image data generated in real time. Further, the cameradata and the radar data generated in real time are used in real time forbasic training of the artificial intelligence modules and may be used inreal time for additional training of the artificial intelligencemodules.

When the fusion data are generated by the fusion data generation unit120, the training unit 140 may additionally input the fusion data intothe artificial intelligence radar module and the artificial intelligencecamera module to additionally train the artificial intelligence radarmodule and the artificial intelligence camera module.

An object detection result obtained by the radar apparatus 10 is morestable than an object detection result obtained by the camera in badweather or at night time. Also, the radar apparatus 10 is capable oflimiting and classifying an object, which is detected as moving on aroad at a predetermined speed or more based on ground speed information,into a specific class. Therefore, it is possible to use the radar datafor training of the artificial intelligence camera module in bad weatheror at night time.

The fusion data used for training may include, for example, vertical andhorizontal location information of the detected object and object typeinformation (for example, vehicle, truck, motorcycle, bicycle,pedestrian, etc.) classified by the previously trained artificialintelligence camera module.

For example, referring to FIG. 4 , the training unit 140 may input thefusion data and radar data (radar data generated based on a reflectedradar signal received in real time) into the artificial intelligenceradar module 22 and additionally train the artificial intelligence radarmodule 22 so that the artificial intelligence radar module 22 can derivea classification result value of the object. Further, the training unit140 may input the fusion data and real-time image data into theartificial intelligence camera module 20 and additionally train theartificial intelligence camera module so that the artificialintelligence camera module 20 can derive camera data including adetection result value of the object and a classification result valueof the object.

Then, when the real-time image data are input into the additionallytrained artificial intelligence camera module, real-time camera data maybe derived by the artificial intelligence camera module.

That is, according to the present disclosure, it is possible toprimarily train the artificial intelligence modules based on radar dataand camera data serving as basic data. This is basic training for theartificial intelligence modules to have minimum performance. Further,according to the present disclosure, it is possible to secondarily trainthe artificial intelligence modules based on fusion data generated tocomplement the disadvantages of the radar data and the camera data, and,thus, it is possible to improve performance of the artificialintelligence modules. Particularly, according to the present disclosure,it is possible to effectively train the artificial intelligence radarmodule by suggesting a fusion data-based classification training methodfor radar data which it is difficult to annotate.

The classification unit 130 may classify the object detected by theradar apparatus 10 through the artificial intelligence modules trainedbased on the generated fusion data. Specifically, the classificationunit 130 may classify the object detected by the radar apparatus 10through the additionally trained artificial intelligence radar module.

Meanwhile, a performance measurement unit (not shown) may measureperformance of the additionally trained artificial intelligence radarmodule based on a classification result value of the object derivedthrough the additionally trained artificial intelligence radar module.

When a performance value of the artificial intelligence radar moduleexceeds a predetermined threshold value, the fusion data generation unit120 may generate fusion data further based on the classification resultvalue of the object derived through the artificial intelligence radarmodule.

For example, referring to FIG. 6 , when the performance value of theartificial intelligence radar module exceeds the predetermined thresholdvalue, the fusion data generation unit 120 may generate fusion databased on the classification result value of the object derived throughthe artificial intelligence radar module 22 and the camera data derivedthrough the artificial intelligence camera module 20.

When the fusion data are generated further based on the classificationresult value of the object derived through the artificial intelligenceradar module, the training unit 140 may additionally input the fusiondata into the artificial intelligence radar module and the artificialintelligence camera module to additionally train the artificialintelligence radar module and the artificial intelligence camera module.

The autonomous driving unit 150 may perform autonomous driving of thevehicle based on the classification result value of the object derivedthrough the artificial intelligence radar module. For example, if avehicle equipped with the radar apparatus 10 drives without a camera,the vehicle may transmit and receive a radar signal through the radarapparatus 10, input a radar signal reflected from an object into theartificial intelligence radar module and perform autonomous drivingbased on a classification result value of the object derived through theartificial intelligence radar module.

Meanwhile, it would be understood by a person with ordinary skill in theart that each of the transceiver 100, the signal processing unit 110,the fusion data generation unit 120, the classification unit 130, thetraining unit 140 and the autonomous driving unit 150 can be implementedseparately or in combination with one another.

FIG. 7 is a flowchart showing a method for classifying an object by theradar apparatus 10 installed in a vehicle according to an embodiment ofthe present disclosure.

Referring to FIG. 7 , in a process 5701, the radar apparatus 10 maytransmit a radar signal to the outside of a vehicle.

In a process 5703, the radar apparatus 10 may receive a radar signalreflected from an object.

In a process 5705, the radar apparatus 10 may process the reflectedradar signal to detect the object.

In a process 5707, the radar apparatus 10 may generate fusion data basedon radar data and camera data.

In a process 5709, the radar apparatus 10 may classify the detectedobject through artificial intelligence modules trained based on thegenerated fusion data.

In the descriptions above, the processes 5701 to 5709 may be dividedinto additional processes or combined into fewer processes depending onan embodiment. In addition, some of the processes may be omitted and thesequence of the processes may be changed if necessary.

A computer-readable medium can be any usable medium which can beaccessed by the computer and includes all volatile/non-volatile andremovable/non-removable media. Further, the computer-readable medium mayinclude all computer storage and communication media. The computerstorage medium includes all volatile/non-volatile andremovable/non-removable media embodied by a certain method or technologyfor storing information such as computer-readable instruction code, adata structure, a program module or other data. The communication mediumtypically includes the computer-readable instruction code, the datastructure, the program module, or other data of a modulated data signalsuch as a carrier wave, or other transmission mechanism, and includes acertain information transmission medium.

The above description of the present disclosure is provided for thepurpose of illustration, and it would be understood by those skilled inthe art that various changes and modifications may be made withoutchanging technical conception and essential features of the presentdisclosure. Thus, it is clear that the above-described embodiments areillustrative in all aspects and do not limit the present disclosure. Forexample, each component described to be of a single type can beimplemented in a distributed manner. Likewise, components described tobe distributed can be implemented in a combined manner.

The scope of the present disclosure is defined by the following claimsrather than by the detailed description of the embodiment. It shall beunderstood that all modifications and embodiments conceived from themeaning and scope of the claims and their equivalents are included inthe scope of the present disclosure.

What is claimed is:
 1. A radar apparatus installed in a vehicle,comprising: a transceiver that transmits a radar signal to an outside ofthe vehicle and receives a radar signal reflected from an object; asignal processing unit that processes the reflected radar signal todetect the object; a fusion data generation unit that projects cameradata to a radar coordinate system and matches the camera data projectedto the radar coordinate system and radar data to generate fusion data;and a classification unit that classifies the detected object using anartificial intelligence module trained based on the generated fusiondata, wherein the radar data include data derived from the reflectedradar signal, and wherein the camera data include data derived fromimage data generated by a camera.
 2. The radar apparatus of claim 1,wherein the signal processing unit is further configured to: perform asignal processing to the reflected radar signal to detect the object,and recognize the object based on a detection result value of theobject.
 3. The radar apparatus of claim 2, wherein the radar datainclude at least one of a fast Fourier transform value of the reflectedradar signal, the detection result value of the object and a recognitionresult value of the object, and the camera data include a detectionresult value of another object included in image data generated by thecamera and a classification result value of said another object.
 4. Theradar apparatus of claim 3, wherein the fusion data generation unit isfurther configured to match the camera data and the radar data for eachtarget to generate the fusion data.
 5. The radar apparatus of claim 4,further comprising: a training unit that: inputs the radar data into anartificial intelligence radar module of the radar apparatus to train theartificial intelligence radar module, and inputs the image data into anartificial intelligence camera module of the camera to train theartificial intelligence camera module, wherein the camera data arederived through the artificial intelligence camera module.
 6. The radarapparatus of claim 5, wherein the training unit is further configured toadditionally input the fusion data into the artificial intelligenceradar module and the artificial intelligence camera module toadditionally train the artificial intelligence radar module and theartificial intelligence camera module.
 7. The radar apparatus of claim6, wherein when a performance evaluation value of the artificialintelligence radar module exceeds a predetermined threshold value, thefusion data generation unit is further configured to generate the fusiondata further based on a classification result value of the objectderived through the artificial intelligence radar module.
 8. The radarapparatus of claim 6, wherein the classification unit is furtherconfigured to classify the detected object through the additionallytrained artificial intelligence radar module.
 9. The radar apparatus ofclaim 3, wherein the radar signal is received while the vehicle isdriving, and the image data are generated while the vehicle equippedwith the camera is driving.
 10. The radar apparatus of claim 9, whereinthe fusion data generation unit is further configured to generate thefusion data by analyzing driving condition of the vehicle while thevehicle is driving and giving weightings to the camera data and theradar data based on the analyzed driving condition.
 11. /The radarapparatus of claim 1, further comprising: an autonomous driving unitthat performs autonomous driving of the vehicle based on aclassification result value of the object.
 12. A method for classifyingan object by a radar apparatus installed in a vehicle, the methodcomprising: transmitting a radar signal to an outside of the vehicle;receiving a radar signal reflected from an object; processing thereflected radar signal to detect the object; matching the camera dataprojected to the radar coordinate system and radar data to generatefusion data based on the radar data and the camera data; and classifyingthe detected object using an artificial intelligence module trainedbased on the generated fusion data, wherein the radar data include dataderived from the reflected radar signal, and wherein the camera datainclude data derived from image data generated by a camera.